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Abstract 

We present a formalism for the description of osciUations in matter of solar, 
atmospheric and long-baseline neutrinos in the two schemes of four-neutrino 
mixing that are allowed by the results of all existing neutrino oscillation ex- 
periments. 
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I. INTRODUCTION 



The evidence in favor of neutrino oscillations found in atmospheric and solar 
neutrino experiments and in the LSND accelerator neutrino experiment 0] implies the ex- 
istence of at least three independent neutrino mass-squared differences (see for a simple 
proof): 

Am2^„ ~ 10-1° eV^ (VO) or Am^^^ ~ IQ-^ - IQ-^eV^ (MSW) , (1.1) 

Aml^^lO-'-10-'eV\ (1.2) 
Aml^^^r^leV. (1.3) 

The two possibilities (see 0) for the solar mass-squared difference Am'^^^ correspond to the 
vacuum oscillation (VO) solution and to the MSW effect respectively. 

The three independent Am^'s in Eqs. (^TT[)-(|OD imply the existence of at least four 
massive neutrinos. In this paper we consider the minimal possibility of four massive neutrinos 
P-16|. In this case, in addition to the three active neutrinos z/g, t'^, Ur in the flavor basis, 



there is a sterile neutrino z/^ that does not take part in standard weak interactions (see [0). 

The results of solar and atmospheric neutrino experiments have been analyzed by several 
groups (see [|6tp], |l8| , p!9| and references therein) under the assumption of transitions into active 



neutrinos (z/g — > z/^,r for solar neutrinos and Vr or z/^ ^ v^. for atmospheric neutrinos) 

or into sterile neutrinos ive for solar neutrinos and z/^ — > z/^ for atmospheric neutrinos). 
However, in the case of four-neutrino mixing transitions into active and sterile neutrinos can 
take place simultaneously for both solar and atmospheric neutrinos. In this paper we present 
a formalism for the calculation of the transition probabilities into active and sterile neutrinos 
of solar z/g's and atmospheric z/^'s, taking into account the matter effects in the Sun and in 
the Earth (see ppj-p^). The derivation of this formalism follows a method similar to the 
one used to develop the formalism that describes neutrino oscillations in the framework of 
three-neutrino mixing [^,^,^^-^|. 



It has been shown |Tl|j31|JT^ that there are only two schemes with four-neutrino mixing 
that can accommodate the results of all neutrino oscillation experiments: 

atm sun sun atm 

(A) mi < m2 ^ m3 < m4 , (B) mi < m2 ^ m^ < m4 . (1-4) 

^ V " ^ ^ ' 

LSND LSND 

These two spectra are characterized by the presence of two pairs of close masses separated 
by a gap of about 1 eV which provides the mass-squared difference AmLgN^ = f^l — 
responsible for the oscillations observed in the LSND experiment. In the scheme A, we have 
^^atm — ^2 ~ ^1 ^"^sun — ^\ ~ ''^35 whcrcas in scheme B, Am^^^^j = m\ — m| and 
Am^^jj = m\ — mi. The phenomenology of neutrino oscillations in these two schemes is 
identical. Hence, for simplicity we shall consider in the following only the scheme B, but all 
the results are also valid in the scheme A and can be obtained by the exchange 1, 2 3, 4 
of the mass eigenstate indices. The mass spectrum is characterized by the hierarchy 

A"^sun = ^21 < Am^,^ = A43 < Am^s^D = ^41 , (1.5) 
with Afcj = m^ — rn?y 
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It has been shown in [|T2| , p!5| that the upper bound N^^^ < 4 for the effective number of 
neutrinos in Big-Bang Nucleosynthesis imphes that jt^ssp + |?/s4p in the scheme B is very 
small. In this case, atmospheric neutrinos oscillate only in the z/^ z/^ channel and solar 
neutrino oscillate only in the Vg channel. However, the value of the upper bound for 

N^^^ is rather controversial and some authors argue that N^^^ could be larger than 
four. Moreover, a large value of |f/s3p + |t/s4p may be compatible with the upper bound 
jyBBN < 4 if a large lepton asymmetry is generated by Vr ^ oscillations at times much 
earlier than the time of Big-Bang Nucleosynthesis [p3| -|35[|. In this paper we do not put any 
constraint on the possible value of |f/s3p + |?7s4p. A global analysis of all neutrino oscillation 
data with the formalism presented here could lead to the determination of the values of If/^sl 
and \Usa\-, which regulate the neutrino oscillations occurring in the early universe. 

The plan of the paper is as follows: in Section || we introduce the general formulas for 
the description of neutrino oscillations in matter in four-neutrino schemes; in Section |T| we 
derive the survival probability of solar i/e's and the probabilities of u^. — >■ and Uf. — > z/^ ,- 
transitions in the Sun; in Section |I^ we discuss the regeneration of solar i/g's in the Earth; 
in Section |V| we present the formalism for the description of the oscillations of high-energy 
atmospheric neutrinos in the Earth; in Section we derive the oscillation probabilities of 
neutrinos in long-baseline experiments; in Section |V11| we draw our conclusions. 



II. MATTER EFFECTS 

In a four-neutrino mixing scheme, the flavor neutrino fields VaL = e, s, /x, r) are related 
to the fields v^l of neutrinos with masses by the relation 

4 

v^L = ^UakVkL (a = e, s,/i,r) , (2.1) 
fe=i 

where [/ is a 4x4 unitary mixing matrix. Neglecting possible CP phases, any 4x4 mixing 
matrix can be written product of six rotations, 

Ul2, Uu, f/l4, U23, f/34, (2.2) 

where 

{Uij)ab = Sab + (cOS^9y - 1) {6iaSib + SjaSjb) + Sm{}ij {6iaSjb - SjaSib) . (2.3) 

The order of the product of the matrices ( |2.2| ) corresponds to a specific parameterization of 
the mixing matrix U. We will see that in order to study matter effects in the oscillations of 
the solar and atmospheric neutrinos, it is convenient to use different parameterizations of 
the mixing matrix U. 

The evolution of the amplitudes ipa (« = e, s, fi, r) of the neutrino flavor eigenstates {ua) 
in matter is governed by the MSW equation 

iA$=7^^^ (2.4) 



3 



where 

Vl/ = (^e,^.,V'/.,^r)^ (2.5) 

(for example, apart from a phase, |z/e) ^— (1, 0, 0, 0)^, etc.) and Ti is the effective Hamiltonian 

H = ^^{M' + A). (2.6) 

Here p is the neutrino momentum, is the mass-squared matrix in the flavor basis, 

M'^ = UMlU\ (2.7) 
with M.Q = diag(m^, m^^ ml, ml), and A is the matrix 

A = dmg{Acc,-ANc,0,0), (2.8) 



with Acc = 2pVc7c and Aj^c = '^pVnc (see [pO|-p^). Here Vcc and V^c are the charged- 
current and neutral-current matter potentials 

Vcc = V2GpNe = 7.63 x 10"^^ ( ^ eV , (2.9) 

\ Na cm"-* / 

Vnc = ~V2GpNn, (2.10) 

where Gp is the Fermi constant, A^^ is the Avogadro number and Nf, and Nn are the 
electron and neutron number densities, respectively. These number densities are given by 
Ne = pN jiXe and Nn = pNa{1 — Ye), where p is the matter density in g/cm^ and is the 
electron number fraction. In the case of antineutrinos Acc and Ajyc must be replaced by 
Acc = ~Acc and A^c = —Anc: respectively. In Eq. ( P^ ) we have subtracted a common 
phase generated hj p + Vjyc that has no effect on flavor transitions. 
The flavor basis (|2.5| ) is related to the mass basis 



Vl/0 = (V>i,^2,^3,^4)^ (2.11) 

by 

^ = [/^°. (2.12) 

Here ipk {k = 1,2,3,4) are the amplitudes of the mass eigenstate neutrinos |z/fc), which are 
related to the flavor eigenstates by 

4 

= ^f/„fc |z/fc) (a = e, s,/i,r) . (2.13) 

k=l 
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III. SOLAR NEUTRINOS 



Because the solar neutrino oscillations are due to the mass-squared difference A21, a 
convenient parameterization of the mixing matrix U for studying the matter effects is 

U = UuU2,U23UuU,3Uu. (3.1) 

Furthermore, it is convenient to work in the rotated basis 

^' = f/''^ ^ , (3.2) 

with 

U' = UuU2,U23UuU,s (3.3) 

and 

r = {^P[,^'„^P'„^P',f. (3.4) 

Here ip^ {k = 1, 2, 3, 4) are the amplitudes of the rotated neutrino states |z/^) = J2a ^akWa)- 
The relation between the rotated basis ( p.2[ ) and the mass basis ( p^.ll| ) is 

^' = U^2 ^° , (3.5) 

from which it is clear that ip'^ = 1/J3 and ip^ = 
The evolution equation for \Ef' is 

iA^' = H'^', (3.6) 

dx 

with 

^' = ^ (f/12 Ml + U'^ A U') . (3.7) 

Because the rotation [734 does not have any effect on the matrix A, the effective Hamiltonian 
Ti' is independent of the mixing angle On the other hand, since the matrix A is not 
invariant under the rotations f/13, f/14, U23, U24,, the general form of H' is very complicated. 
In order to seek a simpler form, we use the fact that the negative results of short-baseline 
Ue disappearance experiments lead to small mixing angles i^i^ and ■&14 |llO|JTT[| . Indeed, for 
A41 in the LSND allowed region, the results of the Bugey z/g disappearance experiment 
imply that 

sm^^i3 + sm^^u = \Ue3\^ + lUe^l^ < 10-2. (3.8) 

Therefore, in the following we neglect the two small angles -^i^ and ^^14 whose contribution 
to the solar neutrino transitions is negligible. Let us, however, emphasize that although 
and 'di4 are both small, at least one of the two must be different from zero in order to 
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generate the z/^j V(. and z/^ 

amplitude is = 4 

With the approximation t9i3 
and its exphcit form is 



z/g oscillations observed in the LSND experiment, whose 

2 



^14 



10,11 



0, the mixing matrix reduces to f/ = f/34 f/24 ?723 f/; 
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U 



Stfl2('=''23'*''24*'»34+^''23'=''34) -'^'»12 (^''23 ^'»34 +^'»23 ''''24 ^'»34 ) '^''23 '=''34 "'^''23 ^''24 *''34 
\ S^12('=''23'''»24'='»34~'''»23'^''34) '=''12 ('^''23 ''''34 "'=''23 ^'»24 '=''34 ) " ('=''23 ^''34 +'^''23 ''''24 '=''34 ) '=''24 '=''34 / 





Si923 '=''24 
"34~*''23^''24"''34 





S^24 
"24 ^''34 



\ 



(3.9) 

with = cos-i? and = sin-i?. One can easily see that the mixing matrix ( p. 91) entails the 
following limiting cases: 

''923 = ''924 = 0. The mixing matrix V assumes a block-diagonal form in which z/g, Vg are 
linear combinations of z/i, V2 alone and z/^, are linear combinations of z/3, z/4 alone. 
Therefore, this case corresponds to pure z/g — > z^s transitions for the solar neutrinos 
and pure z/^ — z/,- transitions for the atmospheric neutrinos. 



''923 = 7r/2. In this case, Us\ = Us2 = and the solar neutrinos undergo z/g 
z/g Ur transitions, whereas the atmospheric neutrinos undergo z/^ 
u., — > Ut transitions. 



Ufj, and/or 
Us and/or 



''^23 = 7r/2, "(934 = tt/2. The mixing matrix U assumes a block-diagonal form in which z/g, Ut 
are linear combinations of z/i, z/2 alone and z/^, Us are linear combinations of z/3, z/4 
alone. This case corresponds to pure z/g ^ z/^ transitions for the solar neutrinos and 
pure z/^ ^ Us transitions for the atmospheric neutrinos. 

^24 = 7r/2. This case is excluded because it yields Us4, = 1. Hence, z/4 = Us is decoupled and 
does not participate in neutrino oscillations, excluding the possibility to explain the 
atmospheric neutrino data. 



With the approximation 1913 = i^i^ = the matrix U' is given by 

U' = UuU24U23, 

and the effective Hamiltonian Ti' assumes the simplified form 



(3.10) 



n' 



which exphcitly reads as 



2p 



^12 



f/i3f/]4^f/24t/23 



/ S21-A21C2tf,,+2AcC 



n' = — 



1 

4p 



12 

A21S2i)i2 






A21S21J 



E21+A21C2^, 



12 

,2 , 
■''23 ''24- 



(3.11) 



-52^23 '=^24^^'^ 



-C^23*2iJ24^-'VC 



-82^23 '=1^24 
-Cd9Q«2iJ9i^ArC 



2™3-2'^^23'=^24^'^C: 



S2i>24-4]VC 



-Si)23S2,524^^C 2m|-2s|_^^Ajvc / 



(3.12) 
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with S21 = 1712 + ^1- This is our desired simpler form of Ti.'. 

From the hierarchy (|1.5| ) and the values of the first two diagonal elements in Ti', we find 
that there is a MSW resonance in the Sun for 

A = A21 cos 2^912 , (3.13) 

with 

A = Acc + COS^ ^^23 COS^ ^924 Amc ■ (3.14) 



The resonance condition ( p. 13 ) has the same form as the well-known resonance condition 



for two-generation mixing, with the replacement of the usual quantity Aqc in the case 
of Ve — ^ t'/^.T transitions, or Acc + Aj^c in the case of z/g — > transitions, by Acc + 



COS §2?. COS 'd24:ANc- Indeed, as explained after Eq. (p.9|), the two extreme cases of Ue ^1 
and Ue — »• i>s transitions for the solar neutrinos are recovered in the four-neutrino mixing 
formalism for '(?23 = vr/2 and ^^23 = '^24: = 0, respectively. 

Since A41 ^ A43 ^ A21 ~ A, the evolution in the Sun of the amplitudes ip'^ and 
ip'^ is adiabatic and decoupled from the evolution of the amplitudes ip'^ and T/'g; which can 
undergo a MSW resonance. To lowest order in the power expansion in the small quantities 
\Acc\/ml ~ \Acc\/ml ~ lANcl/mj ~ lAj^cl/ml ~ Asi/m^ ~ A2i/m^, the eigenvalues of 
the effective Hamiltonian Ti' are 

= ^ (^21 + Acc - COS^ ^^23 COS^ ^924^7VC ± l/ (A21 COS 2^9i2 - Af + (A21 siu 2l?i2)'^ , 

(3.15) 



and = E3 = m|/2p, E^"^ = E4 = m\/2p. The accuracy of Eq. ( |3.15|) is shown in Fig. 
m in which the solid lines show the eigenvalues of the effective Hamiltonian Ti' as functions 
of the electron number density Ne obtained with a numerical diagonalization and the open 
squares show the eigenvalues obtained with Eq. ( p.l5| ). The values of the mixing parameters 
used in drawing Fig. |I] are mi = 0, A21 = 1.0 x 10"^ eV^ A43 = 1.3 x lO'^ eV^ A41 = 1 eV^ 
sin^2'(9i2 = 10~^, cos^'(923 = cos'^ ^2^ = 1/^2, P = 1 MeV and we have assumed Nn = Ne/2. 
The dashed curves in Fig. |I| show the eigenvalues for cos '(923 = 0, corresponding to pure 
— *• i^fj,,T transitions in the Sun. On the other hand, the dotted curves in Fig. ^ show the 
eigenvalues for cos'i923 = cos ■(924 = 1, corresponding to pure Ue — > I's transitions in the Sun. 
One can clearly see that as cos^ {^23 cos^ '(924 increases, the location of the resonance is shifted 
towards higher matter densities. 

Apart from the common term p + V^c which has been subtracted from the beginning, 
the eigenvalues E^ {k = 1, 2, 3, 4) are the energies of the energy eigenstates in matter |z/^), 
whose column matrix of amplitudes 

v^^ = «,^f,^3^,0^ (3.16) 

is given by 

^M^^Mt^,^ (3.17) 
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where 



iUfi)ab = 5ab + (cOS^f^ - 1) i5,aSlb + 52a52b) + sill {5iaS2b ' ^Sa^lfe) (3.18) 

is the orthogonal matrix that diagonahzes the 1-2 sector of the effective Hamiltonian Ti' in 
Eq. (|3.12|) . The value of the effective mixing angle in matter 'd'^^ is given by 

tan2'(9i2 

tan 2<^ = — . 3. 19 

1 - A/A2iCos2?9i2 ^ ^ 



One can clearly see that there is a resonance when the condition ( p.l3| ) is satisfied and the ef- 
fective mixing angle becomes i?^|_R = vr/4. The adiabaticity of the resonance is characterized 
by the adiabaticity parameter (see |2^ and references therein) 
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2 |d<|/dx 



^-^^^'2^- , (3.20) 



R 2pcos2?9i2 



R 



where A = dA/dx and the subscript R denotes values at the resonance point. 

The connection between the flavor basis \E' and the energy basis in matter \E'*^ is given 

by 

^ = f/' f/f^ ^^^^ = u^^J , (3.21) 

where U'^ = U' is the effective mixing matrix in matter given by Eq. ( |3.9| ) with the 
replacement of the vacuum mixing angle •&i2 with the effective mixing angle in matter -d^. 
Therefore, the mixing angles '^23, t?24 and i?34 are not modified in matter, whereas the mixing 
angle ^12 assumes the effective value 'd^ given by Eq. ( |3.19|) . 

The energy basis in matter \E'^ is connected to the mass basis by the relation 

= U^f U12 ^° . (3.22) 

In particular one can see that ip^^ = ips and = ipi. Therefore, the energy eigenstates in 
matter {ly^'^) and {1^4^) coincide with the corresponding mass eigenstates lz/3) and {u^) (energy 
eigenstates in vacuum) and their evolution is adiabatic and decoupled from the evolution of 
the states and \i^2^). 

Let us now calculate the probabilities of flavor transitions for solar neutrinos. We consider 
a neutrino produced in the core of the Sun at the point xo where, for appropriate values 
of A21 and of the neutrino momentum p, the matter density is higher than the resonance 
density given by Eq. ( |3.13|) . Since its state is an electron neutrino state, we have 

^(xo) = (1,0,0,0)^, (3.23) 

and the initial amplitudes of the energy eigenstates are 

(xo) = U^'l (xo) (k = 1, 2, 3, 4) , (3.24) 

i.e. \Ef*^(xo) = (cos^9*2(^o), sin-i^fKxo), 0, 0). The amplitudes of the mass eigenstates |z/fc) at 
the point x in vacuum out of the Sun are 
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i^kix) = ^ (f)k{x - xr) Tkj <pj{xR - xo) U^{xo) {k = 1, 2) 



(3.25) 



i=i,2 



and ipsi^x) = ipAi^) =0. Here xr is the resonance point, (pk{x2 — xi) is the phase that the 
k^^ energy eigenstate acquires going from xi to X2 and Tkj is the amphtude of Uj"^ — > 
transitions in the resonance. The values of the mass eigenstate indices in Eq. (|3.25| ) are 
hmited to 1,2 because ip3{xo) = ip4{xo) = (due to the fact that = U^l = 0), and in 
the resonance there are no transitions from the states and {i'^) to the states and 
1 1/4^), that evolve adiabatically because of the decoupling of the 3-4 sector in the effective 
Hamiltonian ( |3.12|) . 



Since the probabilities of u, 
we have 



M 
3 



z/f and z/f 



,M 



transitions in the resonance are equal. 



\Tl2? = \T2l?=Pc 



\Tii 



221 



(3.26) 



where Pc is the crossing probability given by 



exp (— I 7 -F) — exp ^— 1 7 



sin i9i2 



exp 



(3.27) 



2 ' sin"'-i?i2_ 

where 7 is the adiabaticity parameter (|3.20 ) and F depends on the slope of A in the resonance 



see 



and references therein). For an exponentially varying matter density F = 1 — 
tan^'(9i2. This expression for F is a good approximation for the Sun. 

From Eq. ( p.25| ), the amplitude of the flavor state l^a) at the point x in vacuum out of 
the Sun is 



^a{x) = ^ Uak (pk{x - Xr) Tkj (pjixR - Xq) U^J (xq) , 
k,j=l,2 

and the probabilities of transitions (a = e, s, /x, r) are given by 



(3.28) 



P: 



Sun 



^ Uak (Pkix - Xr) Tkj (pj{xR - Xq) U^- (xq) 



(3.29) 



Because the phases oscillate rapidly over the neutrino energy spectrum and over the spread 
of the source-resonance and resonance-detector distances, the interference terms in this 
expression are practically not observable. The observable averaged probabilities are given 
by 



Sun 



J2 \Uak\'\Tk/\U^;{Xo)\' 
k,j=l,2 

{\Ual\'\U^hxo)\' + |f/.2r|f/e^(Xo)r) (1 - Pc) 
+ {\Ual\'\U^^{Xo)\' + \Ua2\'\U^({Xo)\') Pc. 



(3.30) 



With this expression and Eq. ( p.9|) for the mixing matrix in vacuum and the corresponding 
expression for the effective mixing matrix at the production point Xq (given by Eq. (|3.9|) with 
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the replacement -^u ''^12(3^0)), it is straightforward to calculate the Ue survival probability 
and the probability of Vg transitions: 



P, 



Sun 



cos 2^12 cos2<^(xo) 



P, 



Sun 



COS^ '§23, COS^ 



cos2i^u cos2<^(xo) 



(3.31) 
(3.32) 



One can see that the survival probability has the same form as in the two-generation case (see 
Ml). The probability of u^. v, 



of probability, P^ 



Sun 



P: 



Sun 



transitions is easily obtained from the conservation 
yielding 



^Sun 



P: 



Sun 



(1 - COS^t923 COS^ ■&2A) 



'1 






a-) 


2 ~ 





Pc ) cos2?9i2 COS 219^2 (a;o 



(3.33) 



Figure shows the probabilities ( |3.31| )-( 
Sun as functions of P/A21, with sin^ 2^912 
correspond to cos^ 'd23 cos^ "(924 = (z/g 



.33|) for neutrinos produced in the center of the 
6 X 10~^. The dashed, solid and dotted curves 
transitions only), cos^ i?23 cos^ i?24 = 0.5 and 



cos^ ^923 cos^ -(924 = 1 (^^e — ^ J^s transitions only), respectively. We have used the solar density 
distribution given in ||3^. From Fig. |^a one can see that the survival probability of solar UeS 
is sensitive to the value of cos^ {^23 cos^ {^24 only in the range 4 x 10^ < p/ A21 < 10^ MeV/eV^, 
in which the resonance region is near the center of the Sun and the shift of the resonance 
towards higher matter densities for increasing values of cos^ ^23 cos^ t?24 increases the survival 
probability. 

Let us note that the transition probabilities of solar neutrinos can be calculated more 
accurately by integrating the evolution equation ( |3.6| ) along a realistic solar density distri- 
bution Since the evolution of the amplitudes = ''/'a = V'3 ipl^ = ipi = ip4 is 
decoupled, it is sufficient to integrate the two-dimensional evolution equation 



da; 



^'1 
^2 



1 

4p 



A21S219 



12 

A21C2i,i2-2c2^gC2^_^ylivC 



f2 



(3.34) 



where we have subtracted a common phase generated by S21 that has no effect on flavor 
transitions. 

It is important to note that the evolution equation ( p.34|) and the equivalent probabilities 
( p.31| ), ( p.32|) and ( p.33|) depend only on three parameters, A21, sin 2{}i2 and cos^ ^923 cos^ 't?24, 
i.e. just one more than in the case of two generations which is usually assumed in the analysis 
of solar neutrino data (see and references therein). The two parameters A21 and sin2'i?i2 
are analogous to the usual two-generation mixing parameters Am^ and sin^ 2{}, whereas 
cos^ '§23 cos^ -(924 is a new parameter characteristic of four-neutrino mixing. The existence 
of only one additional parameter with respect to the two-generation case implies that the 
analysis of the solar neutrino data in the four-neutrino scheme is feasible (solar neutrino 
data have already been analyzed in ||2^ in the framework of three-neutrino mixing in which 
there is also one additional parameter compared to the two-generation case). 
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IV. REGENERATION OF SOLAR z^e's IN THE EARTH 



The regeneration of solar electron neutrinos in the Earth has been studied in several 
papers in the two-generation framework (see p0| , |38| -^ and references therein). In this case, 
the survival probability of solar u^s after passing through the Earth is given by the standard 
formula [|0|,|38| 

(l — OP^"'' ( pEarth _ • 2 a\ 
pSun+Earth ^ pSun Ue^UeJ U2-*Ue '^^^^ J / . \ 

where "d is the two-generation mixing angle and P^^^^ is the probability of transitions from 
the vacuum mass eigenstate to v^, in the Earth. One can see that if the matter effect 
inside the Earth is negligible, P^^*^^ is given by |f/e2p = sin^t? and Eq. ( [4.1| ) reduces to 

pSun+Earth pSun 

Let us calculate the expression for the probability Py"^^^^^™^"^ of Vf, Ua transitions 
(a = e, s, fj,, t) of the solar neutrinos after passing through the Earth in the case of four- 
neutrino mixing. We consider different final flavors a because we are interested not only in 
the survival probability of electron neutrinos, but also in the probabilities of z/g — and 
i^e ^n^T transitions. In the four-neutrino scheme under consideration the state describing 
a neutrino reaching the Earth from the Sun is a superposition of the two light vacuum mass 
eigenstates z/i and i'2'- 

ksun) = ^1 l^^l) + ^2 \l'2) , (4.2) 

with the amplitudes ipi and ?/'2 given by Eq. ( |3.25|) . These amplitudes are connected to the 
average probability of z/g Va transitions in the Sun (as those given in Eqs. ( |3.31| ) and 
(S)) by 



Passing through the Earth, the state evolves to 

I^^Sun+Earth) = T ^Sun) = V^l T | Z/i) + ^2) , (4.4) 

where the operator T describes the evolution inside the Earth, including the transitions 
between the states |z/i) and |z/2) induced by the matter effects. The state |i^sun+Earth) describes 
the neutrinos reaching the detector. The probability of Ve — *■ transitions after crossing 
the Earth is given by 

^'"Itf = |(^^aksun+Earth)r = \M^a\r\vr) + ^2 (^^a | T | Z/2) |' . (4.5) 



Since the interference terms are not observable |^6[, the measurable average transition prob- 
ability is 

p!:^it'' = \H'Pu:st + iH'Pu^^t , (4.6) 
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where P^^I}}^^ = \{i>a\T\i'k)f is the probabihty of Uk i^a transitions in the Earth. Since 
the probabihties -P^'^*^ and PR^^^ are connected by the unitarity relation 



PiJi/arth I ptarth i \tt \2 i \tt \2 -r 



(4.7) 



using Eq. ( [4.3| ), the probabihty of 
written as 



transitions after crossing the Earth ( |4.5| ) can be 



pSun+ 



Earth 



pSun _|_ 



(lf/.l 


2 + 


Ua2 


2 


- 2Pj=Z 




C„2 








2 _ 


Ua2 


2 



(4.8) 



This equation is a four- neutrino analogy of the two-generation formula (|4.1| ). Again, one can 
see that if the matter effects inside the Earth are negligible, P^^*^^ = |f/ci2p and Eq. ( |4.8|) 
reduces to -P,|™tr'''^*^ = -^i^e'^i^a • Moreover, the expression in (^) for P^J!!^^^''''*'^ is valid in 
schemes with any number of neutrinos larger than four in which only the smallest mass- 
squared difference contributes to solar neutrino oscillations (rearranging the mass eigenstate 
indices, if necessary). From Eq. ( |4.8| ), for the probabilities of and '^s transitions 

after crossing the Earth we obtain 



Sun+Earth 



and 



Sun+Earth 



P 



pSun I 



P: 



Sun 



(1 - 2P.%J (P, 



Earth 



sm 



^^12) 



cos 2'Qi2 



(4.9) 



(2P, 



Sun 



COS^ -(923 COS^ '624) {Py^^y^ — COS^ 'di2 COS^ ^^23 COS^ '§24) 



COS 2'i9i2 COS^ i?23 COS^ ^24: 



(4.10) 



with the probabilities Py^^^^ and P^^^y^ given by Eqs. ( ^.31] ) and ( |3.32| ), respectively. 
The probability of z/g v^^j. transitions can easily be obtained from the conservation of 
probability, pSun+Earth ^ ^ '_ pSun+Earth _ pSun+Earth_ Qnc cau scc that the z/e survival 

probability ( |4.9| ) has the same form as in the two-generation case (see Eq. ( [4.1| )). 

The expressions ([4.9| ) and ([4.10|) are not defined for cos2'(?i2 = 0. In this case 
psu_n^^ ^ {j^ psu_n^^ ^ cos^ ?923 cos^ ?924/2, as cau be seen from Eqs. ( p3l| ) and ( p32D , 
independently of the values of ^/'i and "^2 in Eq. ([4.2|). In this case, Eq. (|4.3|) does not 



apply and one must calculate l^/'ip = 1 — |'?/'2p using Eq. ( |3.25| ) (or solving numerically the 



evolution equation (|3.34|) in the Sun) and determine Py^^^^^^'^'^ from Eqs. ( |4.6|) and 

The expression ( ^.1CI| ) is also not well defined for cos^'i923 cos^'i?24 = 0. This is due to the 
fact that in this case one has Py'^^y,^ = 0, as can be seen from Eq. (|3.32| ), independently of the 
values of ipi and ■02 in Eq. ( |4.2| ) , and thus Eq. ( [4. 3D does not apply. Since cos^ '^23 cos^ {^24: = 
implies that |?7,3p + |f/,4|2 = 1, the unitarity relation (j^ leads to Py^^'I^l^ = Py^^^I^y^ = 
and we have pSun+Earth ^ q ^^^^ (^. 



The probability Py^^y^ can be calculated by integrating numerically the evolution equa- 



tion (|3.34| ) in the Earth with the initial amplitudes at the point yo of neutrinos entering the 
Earth 



^1(2/0) 
^2(2/0) 



sin 

COS'(9i2 



(4.11) 
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The probability P^J^^^ (y) at the point y is given by 



fc=l,2 



(4.12) 



Another method to calculate the probability P^^^I}^^ is based on the approximation of 
the internal composition of the Earth with two or more shells of density (the minimum two 
shells correspond to the core and the mantle) |2^,^^-^Q. In each shell the corresponding 
energy eigenstates evolve adiabatically, developing a phase, and the amplitudes in confining 
shells are matched on the shell boundaries imposing the continuity of the flavor amplitudes. 
The initial amplitudes in the mass basis are 



^°(yo) = (0,l,0,0)^, 
corresponding to lz/2). The final amplitude in the rotated basis 

^'{yn) = {tp[{yn),^2{yn),^3{yn),i^A{yn)f ■ 



(4.13) 



(4.14) 



after crossing the boundaries of n slabs of matter with constant density at the points yi, y2, 
...,?/„ is 



Ml 



(n) 



tM, 



J (n-l) 



f/'|$(2/2-yi)t/f^ 



Aft 



J (2) 



Uf^y^-yoW^^f Uu^^yo) 
(1) 



(4.15) 



where the notation [. . indicates that the matter-dependent quantities inside of the square 
brackets must be evaluated with the matter density in the i^^ slab that extends from 
to yi. The quantities $(?/) are the diagonal matrices of phases acquired by the energy 
eigenstates in matter: 



$(l/) = diag(e-^^"^e-^^^"^e-^^3'^e-^^"'^) 



(4.16) 



However, because of the structure of the initial amplitudes (|4.13| ) and the fact that at each 
boundary between two slabs only 1/12 operates, the phases of the two decoupled energy 
eigenstates lz/3), |i/4) never come into play in Eq. ( |4.15| ). Therefore, only the 1-2 sector is 
relevant. Defining 



i^'iiyn 
^2(2/" 



^°(l/o) 



^1(2/0) 
^2(2/0) 



<l>(y) = diag(e-^^"^e-*^"^ 



(4.17) 



we have 

^'iyn)= \u^Hyn-yn-i)u^f 



u^2Hy2-yi)u,f 



(n) 
(2) 



?7*|$(y„,_i-l/„_2)t/f^ 



Aft 



U,'^^y,-yo)U,f 



(1) 



(n-l) 

Ui2 ^\yo) 



(4.18) 
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where Uu and are 2x2 matrices obtained from the 1-2 sectors of U12 and f/ii, re- 



spectively. Moreover, since the probabihty of z/2 
by 



transitions in the Earth is given 



Earth 



IV^a(Z/n)P 



k=l,2 



(4.19) 



one can ehminate a common phase in Eq. ([4.18|) . Ehminating the phase generated by the 
hghtest energy eigenstate, the effective matrices of phases $(?/) to be used in Eq. ( ^4.18| ) for 
practical calculations are 



$(1/) =diag(l,e-*^2i'^/2pj ^ 



with 



AM 

^21 



2p (Ef - ) = J(A2i cos2?9i2 - AY + (A21 sm2^ 



12 



(4.20) 



(4.21) 



The formalism presented in this section and in the previous one is appropriate for analyz- 
ing the solar neutrino data in the framework of the four-neutrino scheme B (see Eq. ( |1.4| )) 
and in the scheme A with the exchange 1,2 ^ 3,4 of the mass eigenstate indices. Given 
the correspondence of A21 and sin^ 21912 with the usual two-generation mixing parameters 
Am^ and sin^ 2{}, the oscillations of the solar neutrinos, including matter effects, in the four- 
neutrino schemes depend on only one additional quantity, cos^ 2i?23 cos^ 2'(?24, with respect 
to the two-generation case. Therefore, we believe that the analysis of the solar neutrino data 
in a four-neutrino framework is well within the reach of the groups specializing in this task. 



V. ATMOSPHERIC NEUTRINOS 

The flavor evolution of the atmospheric neutrinos while crossing the Earth depends on 
matter [ P^ , p!^ ,P^-|50| if the matter-induced potential is of the same order or larger than the 



energy difference A43/2P of the two mass eigenstates 1/3, z/4, responsible for the atmospheric 
neutrino oscillations. Since the electron and neutron number densities in the Earth are 
approximately equal and vary from about 2 A^cm~^ in the crust to about 6 A^^cm"^ in 
the center, one can see from Eqs. ( |1.2D and (|2.9|) that matter effects are important for 
atmospheric neutrinos with energy larger than about 10 GeV. 

A convenient parameterization of the mixing matrix U for studying the matter effects 
for the atmospheric neutrinos is 

[7 = ^24^23^14^13^34^12, (5.1) 

with 

{Vij)ab = Sab + {cos ipij - 1) {6ia6ib + 6ja6jb) + SllUpij {6ia6jb " Sja^ib) ■ (5.2) 

Here we have changed the notation for the rotation matrices from Uij to Vij and the notation 
for the mixing angles from t^jj to ipij, in order to emphasize that the parameterization of U 
is different from that used in Sections |T| and |V| for the solar neutrinos. 
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It is convenient to work in the rotated basis 

^' = V'^^, (5.3) 

with 

^' = ^24^23^14^13- (5.4) 

In this case the relation between the rotated basis (|5.3|) and the mass basis (|2.11| ) is 



= V^34V^12^°, (5.5) 

and the evolution equation for \[^' is given by Eq. (|3.6|) , with the effective Hamihonian 

= ^ (Vu Vu Ml V^^ V.^, + V'^ A V'^ . (5.6) 

As in the case of the solar neutrinos, the results of the Bugey disappearance experiment 
PB[ imply the constraint (|3.^), i.e. sin^y^ia + sin^ V9i4 < 10~^. Hence, we can neglect the 
two small angles (pi^ and if 14 whose contributions to the atmospheric neutrino oscillations 
are negligible. In this case, the explicit form of the mixing matrix U = V24 V23 V34 V12 is 



— i>ij3-^2''¥'23 '-■/'12 ¥'23 ¥'24 •'V23 V24 V34~''¥'24 ¥'34 "¥'23 V24 V34 "'"='¥'24 ¥"34 

*¥'12^¥'23 ~'^¥'12*V23 '^V23'^'i=34 '^¥'23*V=34 

\ *¥'12'^¥'23*¥'24 ~'^¥'12 '^¥'23 ^¥'24 "'^¥'23 '^¥'24 '^¥'34 "'^¥'24 ^¥334 "'^¥'23 *¥'24 ^¥=34 +'^¥'24 '^¥'34 / 



(5.7) 



Comparing with the explicit form ( p. 9] ) of the parameterization of the mixing matrix used 
in Sections |ITl| and one can see that (pi2 = ^12, but the other mixing angles are different. 
The mixing matrix (|5.71 ) entails the following limiting cases: 



¥^23 = V^24 = 0. The mixing matrix U assumes a block-diagonal form in which u^, Vg are 
linear combinations of z/i, V2 alone and v^, are linear combinations of 1/3, z/4 alone. 
Therefore, this case corresponds to pure Ug transitions for the solar neutrinos 

and pure z/^ —>■ Vr transitions for the atmospheric neutrinos. 

V^23 = 7i"/2. This case is excluded because it yields = f/^4 = 0, implying that z/^ does 
not mix with the two massive neutrinos 1/3, z/4. Since the mass-squared difference 
m\ — m\ is responsible for atmospheric neutrino oscillations, the disappearance of the 
atmospheric z/^'s cannot be explained. 

f2i = 7r/2. In this case Ugi = Us2 = and the solar neutrinos undergo z/g — > z/^ and/or 
z/g —>■ Ur transitions, whereas the atmospheric neutrinos undergo z/^ — > Ug and/or 
—>■ Ur transitions. 

V?24 = 7r/2, (f23 = 0. The mixing matrix U assumes a block-diagonal form in which z/g, z^r 
are linear combinations of z/i, i>2 alone and z/^, Us are linear combinations of z/3, z/4 
alone. This case corresponds to pure Ur transitions for the solar neutrinos and 

pure z/^ Ug transitions for the atmospheric neutrinos. 
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In the following we present a formalism for the analysis of the oscillations of the atmo- 
spheric neutrinos in the Earth without making any assumption about the value of (/?23, but 
it is clear that in order to explain the disappearance of atmospheric z/^'s observed in atmo- 
spheric neutrino experiments the mixing angle cannot be too large. Indeed, the up-down 
asymmetry of high-energy /x-like events observed in the Super-Kamiokande experiment 



implies the upper bound [IG 



sm V223 



f/Mil' + l?7„2p <0.55 



m2| 



(5.8) 



Moreover, the negative results of the short-baseline CDHS and CCFR accelerator z/^ disap- 
pearance experiments [^,0] imply that |IO| , p 



sm v?23 



t/Mil' + |f/u2p<0.2 



m2| 



for 



(5.9) 



and the limit is much more stringent for some larger values of AmLg^D (for example, |f/^ip + 
f^M2p ^ 2 X 10~^ for AmLgND — 2eV^, see P3). This limit is valid in a large part of the 



LSND 



LSND-allowed range for AmLg^o = Am 



415 



0.2eV2 < Am2g^D<2eV^ 



(5.10) 



With the approximation (^913 = ^914 



V^' = V^24V^23 = 

and the effective Hamiltonian H! is 
which reads explicitly as 



/ 





0, the matrix V is given by 
\ 







"Viz 



^V=23 '''1=24 





^4=23 ^ 



(5.11) 



V^34 VI2 Ml Vl + 4 ^24 ^ V^24 V^23 , , 



(5.12) 



/ S2l-A2lC2^; 



Ap 



V2A, 



cc 



A2lS2<^,i2 



A21S2^12 



S21+A21C2^j2 
-2^23 424^^^^: 



\ 



-52^,23 424^'VC7 



-C^23*2i^24^-'VC 



-S2 



~A43C2, 



^23 ^¥=24^*^ 



V34 



V 



■1=23 V=24 

-C^23 *2<P24 ^NC A43S2¥)34 —■^¥'23 '^2^24 "^NC 



A43 52^34 — Sip23 *2v24 ^iVC 
S43+A43 C2,^34 



(5.13) 



with S43 = ml + m\. 

For high-energy atmospheric neutrinos crossing the Earth S21 ~ A21 ^ \Acc\ ~ 

l^jvcl ~ A43 ^ S43 ~ 2A41 and there is a resonance in the 3-4 sector of the effective 
Hamiltonian ( |5.13|) for 



(sin^ ip2i - sin^ 7^23 cos^ (^24) ^Afc = A43 cos 27234 



(5.14) 
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Since A^c < 0, there can be a resonance for neutrinos if 



|tan(y924| < |sin(y923| and cos 29334 > (5.15) 

or 

|tan(y924| > |sinv323| and cos 2(^)34 < 0. (5.16) 

Otherwise, there can be a resonance for antineutrinos, for which A^c A^c > 0. One can 
see that if cos(/?23 = cosv524 = 1, corresponding to pure atmospheric z/^ transitions, the 
term in parenthesis in Eq. ( ^.14| ) vanishes and there is no resonance for any value of Aj^ci 
for both neutrinos and antineutrinos. Indeed, if cos 9923 = cos 9924 = 1 the matter induced 
terms in the 3-4 sector of the effective Hamiltonian ( |5.13| ) vanish and there is no matter 



effect in atmospheric neutrino oscillations, as expected from the correspondence of this case 
with two-generation z/^ — > transitions that are not affected by matter effects because 
and z/t- feel the same matter potential. 

Since S43 ~ 2A41 ^ A43 ~ \Acc\ ~ I^Afcl ^ ^21 ~ A21, the evolution of the amplitudes 
'i/'^ and '?/'2 is adiabatic and decoupled from the evolution of the amplitudes and iIj'^, that 
are coupled by the matter effects. To lowest order in the power expansion in the small 
quantities |y4cc|/S43 ~ |AArc'|/E43 ~ A43/E43 and neglecting S21 ~ A21, the eigenvalues of 
the effective Hamiltonian Ti' are 

E^^ = - COS^ (p23 COS^ (P2aAnc/2p = - COS^ (p23 COS^ V24VNC , (5-17) 

E^' = Acc/2p = Vcc, (5.18) 

^^24) ^^Nc] + [^43S2<^34 - ■5,^23 -52.^24 Anc?^ ■ (5.19) 

Note that the two light eigenvalues and E^ depend on the matter potentials Vnc and 
Vcc, respectively, but are independent of the neutrino momentum p. The accuracy of the 
expression (|5.19|) for the two heavy eigenvalues E^ and E^ is shown in Fig. |^, in which the 
lines show the third and fourth eigenvalues of the effective Hamiltonian 7i' as functions of the 
neutron number density obtained with a numerical diagonalization and the points along 
the lines show the eigenvalues obtained with Eq. ( p.l9| ). The values of the mixing parameters 
used for drawing Fig. |are mi = 0, A21 = l.OxlQ-^eV^ A43 = 1.3xlO-3eV^ A41 = leV^ 
sin^ 2(y9i2 = 10~2, COSV923 = 1, cos^ ^934 = 0.1. We considered p = 10 GeV and Nn = N^. 
The dashed, solid and dotted lines in Fig. ^ correspond, respectively, to cos^ ip24, = 1, 0.5, 0. 
One can see that for cos^ (p24 = 1 (dashed lines) there is no matter effect. Indeed, since we 
assumed cos 9923 = 1, this case corresponds to pure z/^ — > z/^ transitions. From Fig. ^one can 
also see that the resonance is more pronounced when cos^ 9924 decreases, in agreement with 
the condition ( p.l6[ ) and the fact that the case cos^ ip24, = (dotted lines) corresponds to 



pure z/^ Vs transitions, for which the effect of the matter-induced neutral current potential 
is maximal. 



As in the case of solar neutrinos discussed in Section [IT| , apart from the common term 
P + Vnc that has been subtracted from the beginning, the eigenvalues E^ {k = 1, 2, 3, 4) are 
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the energies of the energy eigenstates in matter jt'^), whose column matrix of amphtudes 
is related to \1>' by 

^^JM = v,f Vif , (5.20) 

where 

{Vif)ab = Sab + (cos (pf^ - l) {6iaSib + 6jaSjb) + sin ipfj {6ia6jb - Sja6ib) {ij = 12, 34) 

(5.21) 

are the orthogonal matrices that diagonalize the 1-2 and 3-4 sectors of the effective Hamil- 
tonian Ti' in Eq. ( ^.13| ). The value of the effective mixing angle in matter is given 
by 

, o M tan 2(^12 

tan 2(^12 = TTT ^ ■ (5-22) 

1 - A/A21 cos 2^912 

However, since A ^ A21, we have 

^f^~7r/2. (5.23) 

Therefore, the mixing in the 1-2 sector is strongly suppressed. The value of the effective 
mixing angle in matter is given by 

, o M A43 sin 2v934 - sin V923 sin 2v924^jvc 

tan 2(^34 = -—^ — ^- — . (5.24) 

A43 cos 2^934 - (sm V524 - sm (^23 cos^ ip2i) A^c 

When the resonance condition ( ^.14|) is fulfilled, the effective mixing angle ip^l is equal to 
7r/4, i.e. the mixing generated by (f^l is maximal. On the other hand, when the equality 
A43sin2(y934 = sin(y923 sin2y924^Afc is satisfied, the mixing is suppressed, as expected from 
the vanishing of the off-diagonal terms in the 3-4 sector of the effective Hamiltonian 
leading to ip^ = if A43 cos 2(^934 > (sin^ ip24 — sin^ (p23 cos^ 1^24) and (p^ = 7r/2 in the 
opposite case. From Eq. (|5.24|) it is clear that if (^23 = V'24 = 0, i.e. there are only — z/^- 
transitions, we have (p^l = ipu, because there is no matter effect, and if ip23 = 0, ip24 = 7r/2, 
i.e. there are only z/^ Vg transitions, we have 

A43 COS 2(^934 - Anc 

which is the usual expression for the effective mixing angle in matter in two-generation v^-Vs 
mixing. 

The connection between the flavor basis \1/ and the energy basis in matter '^'^^ is given 

by 

m = V' Vit = f/^' , (5.26) 

where U^^ = V is the effective mixing matrix in matter which can be obtained 

from the vacuum mixing matrix ( ^.7|) with the replacements ipi2 ^ ti/2 and (^934 1— >■ y9^ (the 
mixing angles (^923 and ip2i are not modified in matter): 
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u 



/ 1 \ 

M 



S^23 '^V23'^^M '^<P23*,^A| 

y ^^23^^24 ^ ~-^V23'^'P24'^^M-C<p24,^^M -^'P23^V24-^<pM+Cy,24'^^M j 



(5.27) 



The amplitudes of the two hght mass eigenstates are related to the amplitudes of the 
flavor eigenstates by the relations 

V^l ~ ~(^^p2Z^^24'^S + "5(^23 V^M ^<4>2Z^^24'^T ■ (5.28) 

< = ^e, (5.29) 

that are independent of the matter density. Furthermore, Eqs. (|5.2CI| ) and ( |5.23| ) imply that 
= —ip2 and ip2^ = independently of the value of the matter density. Therefore, the 
evolution of the amplitudes ipf^ and is adiabatic and decoupled from the evolution of the 
amplitudes ips^ and ipi^ that are coupled by the matter effects. Indeed, the amplitudes 
and ip^ are related to the amplitudes -i/'a and -04 by the relations tp!^ = cos v^lf V's ~ sin ^p^ip'^ 
and ip^ = simp^ip'^ + cos ip^lip'^^, with the mixing angle cp^ which depends on the matter 
density. This means that ^ transitions occur when the variation of the matter 
density is very rapid. 

If a flavor neutrino has been produced at the initial coordinate zq along the neutrino 
trajectory, we have 

i^k^"\^o) = U^il^ (A; = 1,2,3,4), (5.30) 

where we have introduced the superscript (a) to indicate explicitly the initial neutrino flavor. 
The corresponding amplitudes ipf^^°'\z) and iIj2^^°'\z) at the coordinate z are given by 



i;P''\z) = exp / dz U^i {k = 1, 2) , (5.31) 



where the integral must be evaluated along the neutrino path. The elements and 

of the effective mixing matrix in matter ( p.27| ) do not carry the subscript zq because they 

are independent of the matter density, as long as A ^ A21 and Eq. (|5.23|) is satisfied. 

Because of the decoupling of the evolution of the amplitudes ifj'i = ip2 ^ and ^'2 = \ the 
evolution in matter of the amplitudes ip'^ and ip'^^ can be calculated by integrating numerically 
the two-dimensional evolution equation 

i — ( ^'^]= — f -'^«C2,p34-2s2^3c2^4AArc A43S2V34 -8^23 S2^24 ^iVC \ f V's | r^2) 
dx \ i^'l ) 4p \ A43S2¥>34-S,p23'^2,p24^iVC ^4302,^34 -25^ Aatc / \ 4''! J ' 

obtained from the 3-4 sector of (|5.13|) and the definition 

^Pi{z) = e-^«(^-^o) <(^) {k = 3,4), (5.33) 

that eliminates the large common phase generated by S43. For an initial flavor neutrino 
Ua, the amplitudes in the rotated basis at the coordinate zq of neutrino production in the 
atmosphere are given by 
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i^'t\^o)=i^'t\^o) = VU (A: = 3, 4) 



(5.34) 



and we use the notation il)'lf\z)^ ip'l}°'\z) for the the amphtudes in the rotated basis at the 
coordinate z along the neutrino path obtained by solving the differential equations (|5.32|) 
with the initial conditions (|5.34|) . For example, for an initial muon neutrino we have 



COS 





(5.35) 



to be used as the initial condition for the numerical solution of the differential equations 
( p.32| ). The corresponding initial amplitudes %1)^^^\zq) and ip2^''^\zQ) in Eq. (|5.31[ 



are 



jjM 



sm v?23 



TjM 



0. 



(5.36) 



The probability P^^^^^ (z) of z/q, transitions at the point z is given by 



pAtm / N 



fc=l 



(5.37) 



where 'ip^^\z) indicates the amplitude of the flavor state at the coordinate z along the 
neutrino path obtained from a pure flavor state \va) at the initial coordinate 2:0- 

Since in real experiments the interference terms depending on the large phases e~*^«(^~^o) 
are washed out by the averages over the energy resolution of the detector and over the 
production region, the measurable probability is given by 



pAtm 



k=l,2 



+ 



fc=3,4 



(5.38) 



which can be written as 

pAtm / N 



fe=l,2 



fc=3,4 



(5.39) 



Furthermore, using Eq. (^.31|) we obtain 



P, 



Atm 



(^) 



TjMtjM , TjMtjM 



+ 



//(a) , 



fc=3,4 



(5.40) 



with 



E. 



M 



El'' = Vcc + COS^ V^23 cos^ '^2aVi 



NC 



(5.41) 



Since U^i = ?7^[(1 - (5„e), U^i = 6^^ and l/'g = V^^ = 0, from Eq. (|^ one can see 
that the survival probability of atmospheric electron neutrinos is one, independently of the 
matter effect (this is a consequence of the approximation t/13 = Uu = 0): 
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jAtm 



(5.42) 



Therefore 



Atm 



Atm 







The other transition probabihties are given by 



(/?7^e) 



(5.43) 



pAtm / N 



It/, 



M\2\ttM\2 



/3ll 



TJIVI z I 

•^Ql I + 



//(a) , 



A;=3,4 



{a, [3^ e) 



(5.44) 



with the amphtudes ^3*^"'' (-2) and ^'4'""'' {z) obtained from the numerical solution of Eq. ( ^.32| ) 
with the initial condition ( |5.34|) . It is important to note that the elements of the effective 
mixing matrix in matter ( ^.27] ) depend only on the vacuum mixing angles ^523 and ip2A and 
therefore they are independent of the matter density. 

The transition probabilities of atmospheric antineutrinos can be obtained from Eq. (|5.44|) 
changing the signs of the matter potentials. Note that, although we neglected possible 
CP-violating phases in the neutrino mixing matrix, the transition probabilities are not CP- 
symmetric, 



,//(q) 



Atm 



/ pAtm 



(5.45) 



because the matter effect is CP-asymmetric, but they are T-symmetric, 



jAtm 



p. 



Atm 



(5.46) 



because the matter distribution is symmetric along the path of neutrinos crossing the Earth. 

Instead of solving numerically the evolution equation (|5.32| ) for atmospheric neutrinos 
propagating in the Earth, the amphtudes il)l^°'\z) and il)'l^"\z) can be calculated approxi- 
mating the internal composition of the Earth with two or more shells of constant density, 
as done for the regeneration of solar i/g's in Earth in Section In this case, after crossing 
the boundaries of n matter-slabs of constant density at the points zi, • • • ? 2;„ we have 



yM 



Vil^{z,-z,)V,f 



(n) 



(2) 



V^\'^{Zn-l-Zn-2)V.^f 



V,\'^z,-zo)V,t^ 



(1) 



(n-1) 



(5.47) 



where is the 2x2 matrix obtained from the 3-4 sector of and 



11(a) . 



$(2) = diag(l,e-''^«^/2^') 



(5.48) 



with 



[A43 C2<^34 — [s 



2 

1^24 



'^323 ¥'24 



) ^Nc] + [A43S2<^34 — S^23'52(^24^ 



NC\ 



(5.49) 
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The formahsm presented in this section can be apphed to the analysis of the atmospheric 
neutrino data in the framework of the four- neutrino scheme B (see Eq. (|1.4| )) and in the 
scheme A with the exchange 1,2 ^ 3,4 of the mass eigenstate indices. Whereas in the 
case of solar neutrinos the presence of only one additional parameter (cos^ ■i?23 cos^ ■(924) dis- 
tinguishes the oscillation probabilities in the framework of four-neutrino mixing from those 
in two-neutrino mixing, in the case of atmospheric neutrinos there are two relevant mixing 
angles, (p23 and (p24, in addition to the mixing parameters A43 and 9934 corresponding to the 
usual two-generation mixing parameters Am^ and However, information on the values 
of these parameters can be obtained from solar neutrino data (as discussed above, since 
the parameterizations of the mixing matrix U that we have used for the study of solar and 
atmospheric neutrinos are different, we have only the equality (f2i = "^21 and a numerical 
transformation between the two parameterizations is needed in practical calculations in or- 
der to connect the values of the other mixing angles). Therefore, we think that the best 
way to obtain information on four-neutrino mixing from the data of solar and atmospheric 
neutrino experiments is to perform a combined fit. This task is challenging, but far from 
impossible (analyses of the atmospheric neutrino data in the framework of three-neutrino 



mixing taking into account the matter effects have been already carried out [p!9| , p8| -|3 



VI. LONG-BASELINE EXPERIMENTS 

The formalism presented in the previous section for studying the oscillations of the 
atmospheric neutrinos with matter effects can also be applied to calculate the oscillation 
probabilities in long-baseline experiments (LBL) Indeed, the region of the parameter 



space explored by long-baseline neutrino oscillation experiments is similar to that explored 
by atmospheric neutrino experiments. Furthermore, the neutrino beams in long-baseline 
experiments propagate in the mantle of the Earth, where the matter density (pmantie — 
4.5g/cm^) and electron number fraction (Yg ~ 0.5) are approximately constant. Therefore, 
we can apply the formula in Eq. ( |5.47|) with only one slab: 



*"(-)(zi) = V,'f^z^ - zo) V,\'^ ^"^^\zo) , (6.1) 

with all the matter quantities evaluated for an approximate constant density in the mantle. 
Taking into account Eqs. ( p.34| ) and ( |5.48| ), we obtain 



E 

fc=3,4 



Inserting this expression in Eq. (^.40 ), one obtains that the probability of z/q, transi- 
tions with a, /? 7^ e in long-baseline experiments is given by 



^LBL /7-\ _ \tjM\2\ttM\2 
Uc^upK^) — |t^/3ll l^all 



tMttM , ttMttM „,^„ „■ 43-*^ 



2p 



(a,/3^e), (6.3) 



'^/33'-'a3 + '^/34^a4 ^Xp 

where L = zi — zq is the source-detector distance. From the discussion in Section |V| it is 
clear that Pj;'^,, = 1 and P,!"^,, = P,!"^,, = for /3 7^ e. Hence, in the following part of 
this Section we restrict the values of the flavor indices a, P to s, /x, r. 
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The survival probability of can be written as 

P!;r^.AL) = \U^i\' + (1 - V™(L) , (6.4) 



with 



a\ttM\2\ttM\2 /A^'r" 

{\U^^\2+\U^i\2f \ Ap ' 



Defining 



TjM TjM 

COS = = , sin = = , (6.6) 

Eq. (|6.5|) can be written as 

P^:}:t\L) = 1 - sin^ 2e« sin^ , (6.7) 

which shows that P^^]^''^^'^{L) is a two-generation- like survival probability generated by the 
mixing angle ^a- Since P^^ii^^(L) is bounded by 

2 

2o(r ^ dLBL;3,4 



ITJM 


2 _ 


'-^a4 


to 


\TJM 


2 + 


TjM 


2 



= l-sin^2e„<P.t"_i;tW<l, (6.8) 
the survival probability (|6.4|) is restricted to the range 

\u^i\' + mil' - < p!^r^.AL) < if/*(r + (i - lu^m" ■ (6-9) 

In the limit = we have P^^^^^(L) = P^^_}^^f{L), i.e. the survival probability of 
long-baseline u^s reduces to the two-generation form ( |6.7|) . 

Eqs. ( |6.4| ) and ( |0| ) seem to depend on only three parameters, sin^ 2^q, and A4I. 

But since sin^ 2^^ and depend on the neutrino momentum p through the matter-induced 
quantity Aqc, the analysis of long-basehne z/q, disappearance experiments requires one to 
express sin^ 2^^ and in terms of the mixing angles ip23, (p24, ^^34 and A43. Because the 
matter density and composition are known and constant, such an analysis is possible. 

Of particular interest is the survival probability of i/^'s that constitute the main part of 
the initial beam in long-baseline experiments. It is given by 

Pu^XiL) = sin^ ^23 + cos^ ^23 , (6.10) 

where P^^'^^f{L) is given by Eq. ( |6.7| ) with 

^, = <- (6.11) 



If ¥^23 = 0, which is consistent with the experimental bounds ( |5.8| ) and ( |5.9| ), the survival 
probability ( |6.10|) reduces to the two-generation form P^^]^';^''^{L). However, even if if 23 = 0, 
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in general, there are simultaneous z/^ Vr and — > Vg transitions, with pure z/^ — > Ur 
transitions only if y)24 = and pure — > transitions only if Lp2i = vr/2. 
The probabilities of transitions with a ^ (3 are given by 

Pt^^.^L) = 2 \U^i\^ \U^U^ - 4 U^l U^i U^i U^l sin^ j (« ^ /3) . (6.12) 

In particular, the probabilities of v^j, and —>■ Vg transitions are 

Pyf}^ur {P) = \ sin^ 2(^923 sin^ (^24 + cos^ Lp23 sin 2^9^ [(cos^ (f2i - sin^ (^23 sin^ ^^24) sin 2(f^ 



34 



+ sin V923 sin 2^924 cos 2(/}34] sin^ f j , (6.13) 

P^^^^^(L) = - sm 2^923 COS 9924 + COS V523sin2v934 [(sm v924-sin V523COS V524 j sm 2V934 

— sin ip23 sin 2(^924 cos 2(^^] sin^ ( 4^ ) ' (6.14) 
In the limit (p23 = we have 

^i'-^V (L) = cos' ^24 sin^ 2^3*1 sin^ (^) ' ^^'^^^ 

Pl^^.XL) = sin^ ^24 sin^ 2^*1 sin^ f , (6.16) 



4p 

z.e. the — > and z/^ Ug transition probabilities are given by two-generation- 
like expressions weighted by the factors cos'' (p 24 and sin^y924, respectively, resulting in 
pure z/^j oscillations (with = ^934 and A^^ = A43, i.e. without matter ef- 

fects) if ip24 = and pure z/^i Ug oscillations (with (y9|f given by Eq. ( [5.25[ ) and 

= W[A43C2^34 - ANcf + [A43S2<^34]^) if V'24 = vr/2. 



VII. CONCLUSIONS 



We have presented a formalism which is appropriate for analyzing the results of the 
solar, atmospheric and long-baseline experiments in the framework of four-neutrino mixing 
with matter effects. We have considered the two four-neutrino mixing schemes A and B in 
Eq. ( |1.4| ) that are compatible with the results of all the neutrino oscillation experiments 
||TT| , |3llJT6| . Only the scheme B has been considered explicitly here, but all the results are 
also valid in the scheme A and can be obtained with the exchanges 1, 2 3, 4 of the mass 
eigenstate indices. Since the elements Ue3 and f/e4 of the neutrino mixing matrix in the 
scheme B are known to be small (see Eq. ( p.8| )) and negligible for the oscillations of the 
solar and atmospheric neutrinos, we have used the approximation Ue3 = f/e4 = 0. 

For the solar neutrinos the matter effects are important in the case of MSW transitions 
in the Sun (Section |T|) and for the regeneration of z^e's in the Earth (Section fV\j . The 
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oscillations of the solar neutrinos depend only on three parameters, the mass-squared dif- 
ference A21, the mixing angle and the product cos^ '(?23 cos^ "(924 (the parameterization of 
the mixing matrix for the study of solar neutrinos in terms of the mixing angles -^12, "^23, 
■&24 and '(934 is given explicitly in Eq. (|3.9| )). Therefore, only one more parameter than in 
the case of two-neutrino mixing is needed in order to analyze the results of the solar neu- 
trino experiments in the framework of four-neutrino mixing. Since the solar neutrino data 
have already been analyzed in the framework of three-neutrino mixing [^] in which there 
is also one additional parameter compared to the two-generation case, the analysis of solar 
neutrino data in the framework of four-neutrino mixing using the formalism presented here 
is a feasible task. 

As described in Section the oscillations of the atmospheric neutrinos in the Earth 
depend on the mass-squared difference A43 and the three mixing angles ip23, v'24 and ipu 
(the parameterization of the mixing matrix for the study of the atmospheric neutrinos in 
terms of the mixing angles ipu, '^23, ¥^24 and ip^^ is given explicitly in Eq. (p.7|)). Hence, 
in the analysis of the atmospheric neutrino data in the framework of four-neutrino mixing 
there are two parameters more than in the case of two-neutrino mixing. The best way to 
constrain these parameters would be to perform a combined fit of the results of the solar, 
atmospheric and possibly short-baseline and long-baseline neutrino oscillation experiments. 
The formalism presented in this paper can be used to perform this task. 

In conclusion, we would like to note that as a consequence of the approximation 
Ue3 = UeA = the clectrou neutrinos do not oscillate in the atmospheric and long-baseline 
experiments. Because of the upper bound ( p.8|) , this approximation is acceptable for the 
solar and atmospheric experiments, but may be inadequate in the case of high-precision 
long-baseline experiments that could observe transitions below the bound implied 

by the constraint (|3.8D (see [IT^I)- Furthermore, we have neglected the possible presence of 



CP-violating phases in the mixing matrix, because CP violation is not observable in the 
solar neutrino experiments. We note that only the probability of Ue survival and z/g — > z/^,r 
transitions can be measured in the solar neutrino experiments. In the atmospheric neutrino 
experiments detected neutrinos and antineutrinos are not distinguished and the effects of 
the CP-violating phases are washed out by the average over the energy resolution of the 
detector and over the neutrino path length. However, it is possible that the effects of CP 



violation will be measured in high-precision long-baseline experiments WM with conventional 



neutrino beams or with neutrino beams from a neutrino factory In this case, an exact 



formalism of neutrino oscillations with Ue3 7^ 0, 1/^4 7^ and CP- violating phases in the 



neutrino mixing matrix is necessary 53 
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FIG. 1. The solid lines show the eigenvalues of the effective Hamiltonian ( 3.12 ) as functions 
of the electron number density A^e obtained with a numerical diagonalization, whereas the open 
squares show the eigenvalues obtained with Eq. ( 3.15 ). We assumed mi = 0, A21 = 1.0 x 10~^ eV^, 
A43 = 1.3 X 10-3 eV^ A41 = leV^, sin^ 2??i2 = 10"^, cos^ ??23 = cos^ ??24 = l/\/2, p = IMeV 
and Nn = Nf>/2. The dashed and dotted lines show the eigenvalues for cos'!923 = and 
cos7?23 = cos??24 = 1; respectively. 
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FIG. 2. Survival probability of solar v^s (a) and probabilities of Ue t'/i,r (b) and I'e i^s 
(c) transitions in the Sun for neutrinos produced in the center of the Sun as functions of P/A21, for 
sin^ 2-612 = 6 X 10~^. The dashed, solid and dotted curves correspond to cos^ i?23 cos^ i924 = (only 
i^n,T transitions), cos^ t?23 cos^ t?24 = 0.5 and cos^ ■i?23 cos^ t?24 = 1 (only Ug transitions), 

respectively. 
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FIG. 3. The lines show the third and fourth eigenvalues of the effective Hamiltonian ( 5.13 ) as 
functions of the neutron number density A'^ obtained with a numerical diagonalization, whereas the 
points show the eigenvalues obtained with Eq. ( 5.1S| ). We assumed mi = 0, A21 = 1.0 x 10~^eV^, 
A43 = 1.3 X 10-3 eV^, A41 = leV^, sin^ 2^712 = 10" 



COS(/923 = 1, COS^9334 = 0.1, p 



Ne- The dashed, solid and dotted lines correspond, respectively, to cos (^24 



= 10 GeV and 
1, 0.5, 0. 
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